DRAFT 5/5/2008: Estimation of Web Page Change Rates

Search engines strive to maintain a “current” repository of all pages on the web to index for user queries. However, crawling all pages all the time is costly and inefficient: many small websites don’t support that much load and while some pages change very rapidly others don’t change at all. Therefore, estimated frequency of change is often used to decide how often to crawl a page. Here we consider the effectiveness of a Poisson process model for the updates of a page, and the associated Maximum Likelihood Estimator, in a practical setting where new pages are continuously added to the set of rates to be estimated. We demonstrate that applying a prior to pages can significantly improve estimator performance for newly acquired pages.